fix(FR_O): handle new live fields, MQ solar, and historical schema - #8742
Open
VIKTORVAV99 wants to merge 3 commits into
Open
fix(FR_O): handle new live fields, MQ solar, and historical schema#8742VIKTORVAV99 wants to merge 3 commits into
VIKTORVAV99 wants to merge 3 commits into
Conversation
The FR-COR live dataset added aggregate sub-totals (filiere_*) and percentage shares (part_*) that were logged as unknown mode_keys. Ignore these (and date_jour) so they are neither warned about nor double counted. Martinique's live feed now reports photovoltaique; add it to the MQ mapping so solar is no longer dropped. Historical production (target_datetime) was completely broken: it hits a different national dataset whose *_mw schema shares no keys with the live mappings, so every record failed with an empty mix. Add a dedicated historical mapping. The national feed lumps thermal (oil/gas) and bagasse/coal into single values that cannot be split, so both map to the unknown mode. Also guard storage against the frequently-null stockage_mw. Refresh all zone mocks with real samples and add per-zone snapshot tests (SingleFileAmberSnapshotExtension) plus a historical snapshot test; each asserts no warning is logged so unhandled keys are caught. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
stockage_mw is frequently null and tiny when present; treating it as an ignored value avoids needing a special-case null guard on the storage path and keeps the live storage handling unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
stockage_mw is mapped on the historical path as well. The unary minus on production_object[mode_key] crashes when the value is null (which it is for most island records on the national feed), so the negation moves inside the add_value argument and passes None through unchanged - which add_value already handles internally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PaulRoms
reviewed
Jun 9, 2026
| "bioenergies_mw": "biomass", | ||
| "geothermie_mw": "geothermal", | ||
| "thermique_mw": "unknown", | ||
| "bagasse_charbon_mw": "unknown", |
Contributor
There was a problem hiding this comment.
I think bagasse_charbon is a type of biomass. "dry pulpy fibrous material that remains after crushing sugarcane or sorghum stalks to extract their juice. It is used as a biofuel for the production of heat, energy, and electricity, and in the manufacture of pulp and building materials."
PaulRoms
reviewed
Jun 9, 2026
PaulRoms
left a comment
PaulRoms
left a comment
Contributor
There was a problem hiding this comment.
Small comment other LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Several issues surfaced in the FR_O parser (FR-COR, RE, GF, MQ, GP):
Unknown mode_key: 'part_liaisons'warnings (FR-COR) — the live dataset added aggregate sub-totals (filiere_*) and percentage shares (part_*) alongside the per-mode values. They were logged as unknown modes (and risked being double-counted).photovoltaique, but the MQ mapping had no entry for it, so solar was silently discarded and logged as unknown.target_datetimehit a different national dataset (courbe-de-charge-de-la-production-delectricite-par-filiere) whose*_mwschema shares no keys with the live mappings. Every historical record failed withMix is completely empty, returning 0 events plus a flood of warnings.Fix
filiere_*/part_*(via prefix) anddate_jour.photovoltaique → solarto the MQ mapping.*_mwschema, selected whentarget_datetimeis set. The national feed lumps thermal (oil/gas) intothermique_mwand bagasse/coal intobagasse_charbon_mw— neither can be split, so both map to theunknownmode (documented inline). On the historical path the storage value is negated inside theadd_valuecall so the frequently-nullstockage_mwpasses through to the model's internalNonehandling instead of crashing on the unary minus.Tests
FR_GP.json(wrong key) and orphanedFR_COR.json.SingleFileAmberSnapshotExtension.logger.warningis never called, so any unhandled key (new aggregate field or unmapped mode) fails the test. Verified: reverting each fix makes the corresponding test fail.Known follow-up (not in this PR)
Discussed separately as a future improvement: live and historical mappings are not aligned and can't be fully unified — the national (historical) feed is lower-resolution and can't reproduce the live oil/gas/coal split, so historical thermal stays
unknown. Current behaviour is intentionally preserved here. A related pre-existing question — RE's livecharbon→biomass/diesel→biomassmapping — is also left untouched for that future discussion.🤖 Generated with Claude Code